Two fixes for PR #16 round 2 (Codex):

1. P1: Skip quantized caches for non-PP make_cache models.
   ``make_kv_cache`` had a single-node safeguard
   (``_model_is_pipeline_parallel``) on the make_cache-LESS branch,
   but the make_cache-aware branch above it unconditionally replaced
   ``KVCache`` entries with ``QuantizedKVCache`` whenever
   ``EXO_KV_CACHE_BITS`` was set. Models that expose ``make_cache()``
   (e.g. Gemma3 with mixed attention layers) and run single-node
   would therefore crash at runtime with::

       <class 'mlx_lm.models.cache.QuantizedKVCache'> does not yet
       support batching with history

   because mlx-lm's single-node ``BatchGenerator`` calls
   ``_merge_caches`` on every step and ``QuantizedKVCache`` doesn't
   implement ``merge``. Fix: apply the same ``_model_is_pipeline_parallel``
   guard to the make_cache branch and emit the same warning when
   the env var is ignored.

2. P2: Verify peer file integrity before marking download complete.
   ``download_one`` in ``peer_shard_downloader.py`` marked peer
   downloads successful as soon as ``n_read == expected_size``,
   with no content-integrity check. A peer serving wrong bytes
   with the right length (stale/corrupt/malicious) was therefore
   silently accepted as model data, causing hard-to-diagnose
   inference failures or bad outputs.

   Fix: after the peer download completes, fetch the authoritative
   etag/hash from HuggingFace via ``file_meta()`` and validate the
   downloaded file via ``calc_hash``. On mismatch the file is
   removed and the caller falls back to direct HF download. Trusts
   HF as canonical source rather than peer-advertised hash to
   defend against malicious peers that lie about both content and
   hash.

   ``file_meta`` adds one HEAD round-trip per file; ``fetch_file_list_with_cache``
   already requires HF connectivity at this code path so the
   network requirement is not new. The semantics now match the
   direct HuggingFace download path which has done identical
   validation since import.

Address Codex round-2 P1s on PR #16.

P1: Reuse ``ignore_patterns`` when selecting peer download files
- ``download_shard`` (download_utils.py:983) excludes ``original/*`` and
  ``metal/*`` because HuggingFace never downloads them. The peer path
  applied ``allow_patterns`` only, so any repo containing those paths
  (e.g. Llama 3.x) had a required-files list that included files the
  peer never had locally; the strict ``peer_info missing => fail``
  check then aborted the entire transfer and forced a HF fallback for
  every download. Pass the same ``ignore_patterns=["original/*",
  "metal/*"]`` into ``filter_repo_objects`` to match selection.

P1: Honor offline mode in peer file-list fetch
- ``_try_peer_download`` hard-coded ``skip_internet=False`` when calling
  ``fetch_file_list_with_cache``, so offline-configured nodes still
  reached out to HuggingFace before downloading from a LAN peer. On
  cold/offline nodes without a cached file list this raised, the
  ``except: return None`` exited early, and the peer download could
  not even start. Add an ``offline`` parameter to
  ``PeerAwareShardDownloader.__init__`` (defaulting to ``False``) and
  thread the existing ``DownloadCoordinator.offline`` value through
  ``exo_shard_downloader`` so the peer file-list fetch honors the
  same offline contract as ``ResumableShardDownloader``.

Tests
- ``test_offline_flag_defaults_to_false`` / ``..._propagates`` cover
  the constructor wiring directly.
- ``test_try_peer_download_passes_offline_to_fetch_file_list`` patches
  the import binding and asserts ``skip_internet=True`` is forwarded
  when the downloader is constructed with ``offline=True``.
- ``test_try_peer_download_filters_ignore_patterns`` records every
  ``filter_repo_objects`` call and asserts that the peer path
  requested ``ignore_patterns=["original/*", "metal/*"]`` -- the
  exact set ``download_shard`` uses.

… port

Codex P1 (round 3): peer downloads were calling ``file_meta()`` for every
file even when the coordinator ran with ``--offline`` / ``EXO_OFFLINE=true``.
Any failure to reach HF (the entire point of offline mode) was treated as
an integrity-check failure, the peer-fetched bytes were deleted, and the
cold offline node was left with no path to complete model sync. When the
downloader is in offline mode we now trust the LAN peer's bytes (size
already enforced by ``download_file_from_peer``) and skip the HF
canonical-hash call entirely. Online mode still validates against HF.

Codex P2 (round 3): the peer-download listener was hard-coded to a
single module-level constant, so a same-host multi-node deployment
crashed on the second process with ``address already in use``. Add a
``--peer-download-port`` CLI flag (default ``EXO_PEER_DOWNLOAD_PORT``)
and thread it through ``Node`` -> ``Worker`` and ``PeerFileServer``,
replacing the import-time constant. Cluster-wide convention: every
node uses the same value (peer discovery still uses the local value as
the assumed remote port). Cross-node port advertisement via state is a
documented follow-up.

Tests:
- ``TestPeerDownloadIntegrityCheckRespectsOfflineMode`` covers both
  modes: offline must not call ``file_meta`` and must keep the bytes,
  online must still call ``file_meta``.

This addresses two Codex P2 findings on PR #16.

`_resolve_peer_endpoint` returned on the *first* topology edge it
visited for a peer, so when ``out_edges`` happened to yield the
SocketConnection before the RDMAConnection (insertion order is not a
stable contract on the topology graph), the peer was silently labelled
``socket`` and lost its RDMA priority in the peer ordering. The lookup
now scans all edges for a peer, prefers RDMA whenever any RDMA edge
exists (using the companion socket address for the actual TCP connect),
and only falls back to ``socket`` when no RDMA edge is present. The
unused `_find_socket_ip` helper that was meant as a fallback for the
RDMA branch is removed -- its behaviour is folded into the new scan.

`PeerFileServer` was hard-wired to ``EXO_DEFAULT_MODELS_DIR``, but
``select_download_dir_for_shard`` legitimately writes shards across
``EXO_MODELS_DIRS`` (custom paths, low-disk fallback) and we also have
shards available in ``EXO_MODELS_READ_ONLY_DIRS`` mounts. Pre-fix, any
model that landed outside the default directory was invisible to /status
and /files, so peers always fell back to HuggingFace and the new peer
path was a no-op for valid multi-directory deployments. The server now
takes ``models_dirs`` (a sequence) and probes every configured root in
caller-specified priority, checking each candidate against
``_resolve_child`` for path-traversal safety.

Adds regression coverage:
* `test_peer_state.py` covers RDMA-first ordering for both edge insertion
  orders and the socket-only / RDMA-only edge cases.
* `TestPeerFileServerMultipleDirectories` covers serving from a secondary
  writable directory and a read-only mount, plus the constructor's
  empty-list rejection.

Two Codex round-(N+1) P1 fixes for PR #16:

1. peer_download.py:120 -- "Reject oversized stale partials before
   peer resume". The resume loop ran ``while n_read < expected_size``,
   so any pre-existing ``.partial`` larger than ``expected_size``
   skipped the loop entirely and the final ``rename`` accepted the
   bad bytes as the canonical download. In offline mode (where peer
   download integrity check is intentionally skipped) this would
   permanently poison the model cache. Discard the stale oversized
   partial up front and restart from zero on this peer.

2. constants.py:73 -- "Keep node identity keypair out of cache
   storage". ``EXO_NODE_ID_KEYPAIR`` was rooted at ``EXO_CACHE_HOME``,
   which is subject to normal cache cleanup (e.g. ``trash
   ~/.cache/exo``); a wipe silently regenerated a fresh peer ID and
   broke cluster membership / mDNS routes. Move it to
   ``EXO_CONFIG_HOME`` (matching the existing
   ``test_node_id_in_config_dir`` invariant) and add a one-shot
   transparent migrator in ``router._migrate_legacy_node_id_keypair``
   so existing nodes retain identity after the upgrade.

Regression tests:

* ``test_oversized_stale_partial_is_discarded_and_retransferred`` --
  pre-fix would rename the junk ``.partial`` as the downloaded
  weights file; post-fix the file is re-fetched and matches the
  canonical bytes.
* ``test_legacy_keypair_is_migrated_to_new_location``,
  ``test_migration_is_idempotent_when_new_location_already_present``,
  ``test_migration_skipped_when_no_legacy_file``,
  ``test_get_node_id_keypair_uses_migrated_legacy_keypair``.

X-Orchestraitor-Plan: ecosystem_repo_standardization_aeee88ba
X-Agent-Platform: cursor-claude-opus-4.7

Codex P1/P2 (PR #16 round-(N+2), router.py:297, router.py:322):
the same-host multi-node workflow this PR introduces (distinct
peer-download ports per process) requires distinct NodeIds per
process so peer-discovery's self-skip and routing's unique-NodeId
invariants hold. Concurrent startups also raced on the legacy
cache->config keypair migration because it ran before FileLock.

- Add ``process_scope`` parameter to ``get_node_id_keypair`` that
  is folded into the on-disk filename (``node_id.<scope>.keypair``).
  Single-process deployments default to None and keep the existing
  shared file; ``main.py`` passes ``args.peer_download_port`` so
  multi-process same-host runs land on distinct identities.
- Move the legacy migration call inside the ``FileLock`` so two
  processes can't both pass the existence check and race into
  divergent in-memory vs. on-disk identities.
- Legacy file remains unscoped: the first process to migrate
  adopts the operator's existing identity; later processes (other
  scopes) start with fresh keypairs, which is exactly what
  per-process isolation requires.
- Add regression tests covering distinct scopes producing
  distinct keypairs, scope stability across calls, scoped legacy
  adoption, and a structural check that the migration runs inside
  the FileLock.

Codex P1 (PR #16 round-(N+3), main.py:74): the previous scope used
``args.peer_download_port`` only. With ``--no-downloads`` /
``--no-peer-download`` the peer file server doesn't bind, so two
same-host processes can both keep the default
``peer_download_port`` and would then load the same scoped
keypair file -- producing identical ``NodeId``s and breaking
election/routing's unique-NodeId invariants. The new
``_node_id_keypair_scope`` helper combines libp2p, api, and
peer-download ports: at least one of those MUST differ between
two same-host processes (each is a distinct local socket bind),
so the resulting scope is always per-process unique while
remaining stable across restarts of the same configuration.

Codex P1 (PR #16 round-(N+3), peer_download.py:162): on resume
``download_file_from_peer`` sends a ``Range`` header but accepted
HTTP 200 and appended to the existing partial. A non-compliant
peer server is allowed to ignore Range and return full content
with 200, which would duplicate bytes, push ``n_read`` past
``expected_size``, and -- because offline mode skips hash
verification -- silently poison the model cache by renaming the
oversized file as success. Now we treat 200-on-resume as a
restart: discard the partial, reset ``n_read = 0``, and the next
loop iteration re-fetches from zero.

Add regression tests:
- ``TestNodeIdKeypairScope`` covers the per-process scope helper:
  distinct libp2p / api / peer-download ports each yield distinct
  scopes; the same args yield the same scope; and the original
  bug (same default peer_download_port with peer-download
  disabled) is now isolated by libp2p_port differences.
- ``test_resume_with_200_response_discards_partial_and_restarts``
  stands up a tiny aiohttp server that always returns 200 (even
  for ranged requests), primes a partial file, and asserts the
  client discards the partial, restarts from zero, and lands the
  canonical bytes matching ``expected_size``.

… peer responses

main.py: when --libp2p-port 0 is set, the configured value is the
literal 0 even though each process binds a different ephemeral port at
runtime. Two same-host worker-only processes (no API, no peer
download) sharing the default api/peer ports would otherwise produce
identical scope strings and load the same on-disk keypair file,
breaking the unique-NodeId invariant. Fold os.getpid() into the scope
when libp2p_port == 0; the trade-off (ephemeral identity for
ephemeral ports) is the right semantic since the operator opted into
ephemeral binding by setting libp2p_port=0.

peer_download.py: bound the inner read by 'expected_size - n_read' and
treat any extra bytes as a peer protocol violation. Pre-fix the loop
kept appending until EOF and only checked n_read < expected_size
afterwards, so an oversized response (peer serving a stale/wrong blob)
was accepted as success and renamed into the cache. In offline mode
hash verification is skipped, so this silently poisoned local weights.

New tests:
- test_libp2p_port_zero_uses_pid_for_per_process_isolation: verifies
  the scope contains 'pid-<getpid>' when libp2p_port=0.
- test_libp2p_port_zero_in_two_processes_yield_distinct_scopes:
  monkeypatches os.getpid to simulate two same-host processes both
  binding libp2p_port=0 with identical api/peer ports and asserts
  the scopes diverge.
- test_oversized_peer_response_is_rejected_and_restarted: stands up a
  bad aiohttp peer that always serves canonical+'POISONED' bytes and
  asserts the client never lands the trailing junk in the cache.

Squashes 30 commits from feature/drafter-in-process-tuning onto
post-#16 main. Cumulative behavior changes:

- Add Drafter abstraction + n-gram drafting strategy as a non-model
  alternative for speculative decoding (model | pipelined | ngram | none)
- Wire drafter KV prefix cache + manual drafter prefill on the spec path
- Bypass exo prefill+prefix cache for drafters: mlx_lm prefills target
  and drafter natively via speculative_generate_step
- Add drafter benchmark harness (bench/drafter_bench.py) for A/B
  comparing draft modes
- Forward draft_mode override and record n-gram acceptance for adaptive K
- Per-request drafter overrides + warmup with configured K
- Add drafter tuning, builder, and adaptive-K-gate test coverage

Conflict resolution in src/exo/worker/engines/mlx/cache.py
(make_kv_cache):
- Keep #16's PP-gated KV_CACHE_BITS quantization block (operator-facing
  EXO_KV_CACHE_BITS feature for pipeline-parallel deployments) and the
  #16 PP gate in the no-make_cache() branch (prevents the documented
  BatchGenerator + QuantizedKVCache.merge() crash in single-node mode)
- Adopt #19's docstring focusing on mlx_lm.speculative_generate_step
  compatibility
- Drop the unconditional c.step = 16384 mutation on plain KVCache
  instances returned by model.make_cache() so the spec-decoding path
  receives caches whose allocation policy matches each architecture's
  declared shape (consistent with the V2 walk-back rationale in commit
  241fe2c "Use mlx_lm native caches on spec-decoding path")
- Retain qc.step = 16384 on QuantizedKVCache instances we construct
  ourselves in the PP path (internal to that allocation, doesn't affect
  spec-decoding which never reaches that branch)

Squashes 30 commits from feature/drafter-in-process-tuning onto
post-#16 main. Cumulative behavior changes:

- Add Drafter abstraction + n-gram drafting strategy as a non-model
  alternative for speculative decoding (model | pipelined | ngram | none)
- Wire drafter KV prefix cache + manual drafter prefill on the spec path
- Bypass exo prefill+prefix cache for drafters: mlx_lm prefills target
  and drafter natively via speculative_generate_step
- Add drafter benchmark harness (bench/drafter_bench.py) for A/B
  comparing draft modes
- Forward draft_mode override and record n-gram acceptance for adaptive K
- Per-request drafter overrides + warmup with configured K
- Add drafter tuning, builder, and adaptive-K-gate test coverage

Conflict resolution in src/exo/worker/engines/mlx/cache.py
(make_kv_cache):
- Keep #16's PP-gated KV_CACHE_BITS quantization block (operator-facing
  EXO_KV_CACHE_BITS feature for pipeline-parallel deployments) and the
  #16 PP gate in the no-make_cache() branch (prevents the documented
  BatchGenerator + QuantizedKVCache.merge() crash in single-node mode)
- Adopt #19's docstring focusing on mlx_lm.speculative_generate_step
  compatibility
- Drop the unconditional c.step = 16384 mutation on plain KVCache
  instances returned by model.make_cache() so the spec-decoding path
  receives caches whose allocation policy matches each architecture's
  declared shape (consistent with the V2 walk-back rationale in commit
  241fe2c "Use mlx_lm native caches on spec-decoding path")
- Retain qc.step = 16384 on QuantizedKVCache instances we construct
  ourselves in the PP path (internal to that allocation, doesn't affect
  spec-decoding which never reaches that branch)

Co-authored-by: jw-wcv <101585096+jw-wcv@users.noreply.github.com>

Cumulative #20 changes rebuilt onto post-#19 main via 3-way merge of
#20's tip (ec255d6) into team-wcv/main. Replaces the earlier
pure-diff approach which silently reverted ~12 #16 surfaces
(peer_download.py, peer_file_server.py, mdns_announcer.py,
router.py, main.py, constants.py, coordinator.py, etc.) -- all of
those are now preserved at their main state.

57 files changed, +14,662 / -531. No conflict markers.

Codex P1 (remote_drafter.py:673): symmetric drift guard.

The original guard only rejected ``num_forwards > drafts_buffer_size``
(target K > drafter K) and missed the reverse direction. With drafter
K > target K, the drafter would still send ``drafts_buffer_size``
uint32s while the target read only its smaller ``K + 1``, leaving
trailing bytes in the socket and corrupting the next command frame.

Fix carries ``target_drafts_buffer_size`` on every command frame
(slot 7, previously reserved 0), and the drafter validates
``target_drafts_buffer_size == drafts_buffer_size`` BEFORE the
session-cache lookup so a desynced wire fails with a protocol-level
error instead of an incidental "unknown session" caused by garbage
in slot 6. The original ``num_forwards > drafts_buffer_size`` guard
is retained as defense in depth against an out-of-bounds slice in
``_run_drafter_*``.

Codex P2 (batch_generator.py SequentialGenerator.close): explicit
``del self.draft_model`` + ``self.draft_model = None`` before the
existing ``del self.model, self.tokenizer, self.group`` so MLX
reclaims drafter weight buffers on close.

Tests added by this PR:
  test_command_frame_round_trip: parametrized with new buffer field.
  test_command_frame_rejects_target_buffer_out_of_uint32_range: new.
  test_drafter_serve_loop_rejects_reverse_k_drift: regression test
  for the previously-uncovered direction.
  test_placement_drafter_asymmetric.py: 19 placement tests covering
  asymmetric drafter eligibility, RDMA reachability, socket-only
  fallback, and degradation events.

Validation:
  basedpyright: 0 errors, 0 warnings, 0 notes
  ruff check: clean
  ruff format: clean
  pytest: 905 passed, 16 pre-existing test_placement.py failures
    (same on team-wcv/main; not introduced by this PR)

Cumulative #21 changes rebuilt onto post-#20 main (the freshly-rebuilt
clean #20 base) via 3-way merge of #21's tip (628e497) into a fresh
branch off the new clean #20 (71ef3ad). Replaces the earlier
pure-diff approach which sat on top of the regression-laden old #20
and inherited the same silent reverts of #16 surfaces (peer_download,
peer_file_server, mdns_announcer, router, main, constants, coordinator).
All those #16 surfaces are preserved at their main/post-#19 state.

23 files changed, +3,867 / -575. No conflict markers.

Codex P1 fixes (preserved):
  - worker/main.py: _should_drop_generation_task_at_drafter predicate
    guards generation tasks from DrafterRunner.
  - generate.py: clamp_num_draft_tokens_to_transport accepts both
    DrafterTransport and RemoteTransport via HasNumDraftTokens Protocol.
  - placement.py: directional reachability check (drafter -> target
    rank 0) via _drafter_node_is_reachable.
  - utils_mlx.py: mx_all_gather_tasks splits leftover into
    front-of-queue + back-of-queue so admission can't starve a fresh
    peer's delayed first page.
  - pipelined_drafter.py: _get_tokenizer_vocab_size walks four
    fallbacks (added_vocab + len, len, max id, base).
  - instances.py: target_peer_socket_port is Optional with default=None
    for backward compat with replay events.

Production hardening additions:
  - TargetPeerFanout dedicated TCP fanout (sidesteps JACCL int/float
    wire-conflation bug entirely).
  - Backend-aware mx_broadcast_int_list (ring uses all_sum, JACCL
    uses send/recv).
  - DrafterAbortedError + DRAFT_ABORT_SENTINEL + _broadcast_abort
    resilience path.
  - Per-request metrics dict (proposed/accepted/rounds) +
    EXO_SPEC_DIAG-gated _spec_diag hooks.
  - Pre-allocated listener port + reserved_ports set threading so
    drafter/target-peer/jaccl-coordinator all draw distinct ports.
  - Kernel-vetted random_ephemeral_port_excluding(reserved) helper
    layered on the existing kernel-vetted random_ephemeral_port.
  - Spec-decode telemetry on ChatCompletionResponse
    (proposed_draft_tokens, spec_decode_rounds,
    accepted_prediction_tokens, rejected_prediction_tokens).
  - bench_compare.py and bench_concurrent.py for spec-decode
    benchmarking sweeps.

Tests added:
  test_placement_drafter_asymmetric.py:
    test_asymmetric_drafter_and_target_peer_ports_are_distinct
    test_drafter_and_target_peer_avoid_jaccl_coordinator_port
    (both updated with explicit node_rdma_ctl=NodeRdmaCtlStatus(enabled=True)
    setup since post-#19 main's _all_rdma_ctl_enabled filter requires it).
  test_pipelined_drafter.py: TestDrafterAbortRecovery,
    TestGetTokenizerVocabSize (4-fallback walk regression coverage).
  test_remote_drafter.py: resilience tests for sticky _is_failed flag
    + fail-fast open_session check + try/except OSError wrappers.
  test_utils_mlx_broadcast.py: task-ID encoding, _MX_AGREE_MAX_TASKS
    admission, leftover-front/back-of-queue split coverage.
  test_spec_diag_gating.py: EXO_SPEC_DIAG diagnostic toggle.
  bench_compare.py / bench_concurrent.py: '# type: ignore' headers.

Validation:
  basedpyright: 0 errors, 0 warnings, 0 notes.
  ruff check: clean.
  ruff format: clean.
  pytest: 945 passed, 16 pre-existing test_placement.py failures
    (same on team-wcv/main; not introduced by this PR).

…ocket transports (#20)

* Asymmetric pipelined drafter: V1/V2/V3 transports + reverse-K-drift fix

Cumulative #20 changes rebuilt onto post-#19 main via 3-way merge of
#20's tip (ec255d6) into team-wcv/main. Replaces the earlier
pure-diff approach which silently reverted ~12 #16 surfaces
(peer_download.py, peer_file_server.py, mdns_announcer.py,
router.py, main.py, constants.py, coordinator.py, etc.) -- all of
those are now preserved at their main state.

57 files changed, +14,662 / -531. No conflict markers.

Codex P1 (remote_drafter.py:673): symmetric drift guard.

The original guard only rejected ``num_forwards > drafts_buffer_size``
(target K > drafter K) and missed the reverse direction. With drafter
K > target K, the drafter would still send ``drafts_buffer_size``
uint32s while the target read only its smaller ``K + 1``, leaving
trailing bytes in the socket and corrupting the next command frame.

Fix carries ``target_drafts_buffer_size`` on every command frame
(slot 7, previously reserved 0), and the drafter validates
``target_drafts_buffer_size == drafts_buffer_size`` BEFORE the
session-cache lookup so a desynced wire fails with a protocol-level
error instead of an incidental "unknown session" caused by garbage
in slot 6. The original ``num_forwards > drafts_buffer_size`` guard
is retained as defense in depth against an out-of-bounds slice in
``_run_drafter_*``.

Codex P2 (batch_generator.py SequentialGenerator.close): explicit
``del self.draft_model`` + ``self.draft_model = None`` before the
existing ``del self.model, self.tokenizer, self.group`` so MLX
reclaims drafter weight buffers on close.

Tests added by this PR:
  test_command_frame_round_trip: parametrized with new buffer field.
  test_command_frame_rejects_target_buffer_out_of_uint32_range: new.
  test_drafter_serve_loop_rejects_reverse_k_drift: regression test
  for the previously-uncovered direction.
  test_placement_drafter_asymmetric.py: 19 placement tests covering
  asymmetric drafter eligibility, RDMA reachability, socket-only
  fallback, and degradation events.

Validation:
  basedpyright: 0 errors, 0 warnings, 0 notes
  ruff check: clean
  ruff format: clean
  pytest: 905 passed, 16 pre-existing test_placement.py failures
    (same on team-wcv/main; not introduced by this PR)

* Fix 16 pre-existing JACCL placement tests by populating node_rdma_ctl

These tests existed on team-wcv/main as known failures: they call
place_instance(...) without populating the node_rdma_ctl parameter,
which causes the JACCL placement filter _all_rdma_ctl_enabled
(placement.py:309) to reject all RDMA cycles because
rdma_ctl_status.get(node_id) returns None for every node.

Fix: add node_rdma_ctl={node: NodeRdmaCtlStatus(enabled=True), ...}
to each test's place_instance call so the JACCL request fails (or
succeeds) for the right reason -- the Thunderbolt IPv4 preflight
behavior the test was designed to exercise.

No test logic, assertions, or topology setup modified -- only the
missing keyword argument added.

Tests fixed (all in src/exo/master/tests/test_placement.py):
  test_jaccl_placement_skips_thunderbolt_preflight_for_single_node_fallback
  test_jaccl_placement_accepts_maybe_ethernet_thunderbolt_bridge
  test_jaccl_placement_requires_repaired_thunderbolt_ipv4_paths
  test_jaccl_placement_falls_back_to_eligible_cycle_when_another_cycle_has_invalid_path
  test_jaccl_placement_prefers_eligible_cycle_among_multiple_size_2_cycles
  test_jaccl_placement_singleton_fallback_picks_best_node_regardless_of_tb
  test_jaccl_placement_allows_nodes_with_unknown_network_info
  test_jaccl_placement_allows_nodes_with_unclassified_interface_typing
  test_jaccl_placement_still_rejects_nodes_with_known_non_tb_paths
  test_jaccl_placement_rejects_nodes_with_only_loopback_unknown_typing
  test_jaccl_placement_allows_nodes_with_partial_interface_typing
  test_jaccl_placement_allows_bridge0_thunderbolt_with_unknown_typing
  test_jaccl_placement_allows_non_zero_bridge_index_thunderbolt
  test_jaccl_placement_rejects_nodes_with_vm_stack_bridges_and_primary_en
  test_jaccl_placement_rejects_nodes_with_unknown_en0_and_typed_wifi
  test_jaccl_placement_rejects_nodes_with_only_vpn_tunnel_unknown_typing

Validation:
  basedpyright: 0 errors, 0 warnings, 0 notes
  ruff check: clean
  ruff format: clean
  pytest: 961 passed, 1 skipped (was 945 passed + 16 failed)

* Address Codex re-review findings on rebuilt PR #20

Restores one regression that was lost during the 3-way merge rebuild
plus 9 distinct findings from Codex's re-review.

LOST REGRESSION (must restore from OLD #20 = ec255d6):
  - remote_drafter.py: symmetric K-drift guard via
    target_drafts_buffer_size in command frame slot 7. The new clean
    #20 base only had the forward-direction guard (num_forwards >
    drafts_buffer_size) and missed the reverse direction (drafter K >
    target K). target_drafts_buffer_size is now validated BEFORE the
    session-cache lookup in drafter_serve_loop so a desynced wire fails
    with a protocol error instead of an incidental "unknown session".

P1 fixes:
  - placement.py: warm-drafter selection. Two-pass loop -- first pass
    picks an eligible candidate that has drafter weights on disk via
    _node_has_any_drafter_on_disk, second pass falls back to first
    eligible. Fixes FileNotFoundError in DrafterRunner._handle_load
    when first memory-eligible node is cold but a later one is warm.
  - utils_mlx.py: drafter listener bind retry. EADDRINUSE on target
    rank 0 is now retried via _bind_drafter_listener_with_retry
    (budget 8) with a fresh random_ephemeral_port draw each time;
    prior single-shot bind would hang ConnectToGroup on stale listeners.
  - placement.py: drafter-cycle reservation graceful degrade. Probe
    cycles_excluding_drafters against filter_cycles_by_memory first;
    fall back to unfiltered candidates if no reserved cycle has enough
    memory, so we degrade drafter rather than failing the instance.
  - placement.py: single-rank target stays on MlxRing. Pin
    Sharding.Pipeline + InstanceMeta.MlxRing for len(cycle) == 1
    BEFORE the JACCL upgrade gate, so single-node ring + asymmetric
    drafter doesn't trip the JACCL TB-IPv4 preflight.
  - remote_drafter.py: assert OP_FORWARD reply width invariant
    (len(outputs) == num_forwards AND len(outputs) <=
    drafts_buffer_size) before padding so a regressed
    _run_drafter_forwards_remote can't silently desync the wire.

P2 fixes:
  - api.py: false positive (cap is already 1024).
  - worker/main.py: drop path now calls
    runner.mark_task_dropped_locally so the planner doesn't re-select
    the same task every 100 ms tick.
  - generate.py: gate asymmetric drafter session-open on
    asymmetric_drafter_active AND draft_mode != "none" so demoted-
    to-none requests don't open sockets.
  - utils_mlx.py: single-rank asymmetric path now sets
    drafter_id from drafter_placement.drafter_model_id so
    GenerationStats.drafter_model_id is populated for telemetry.

Tests added:
  - test_drafter_serve_loop_rejects_reverse_k_drift: regression test
    for the lost symmetric K-drift guard (drafter K > target K
    direction).
  - test_command_frame_round_trip / test_command_frame_rejects_*:
    coverage for the new target_drafts_buffer_size slot.
  - Two existing asymmetric placement tests renamed/updated to expect
    MlxRingInstance for single-rank targets (the P1.4 fix changes
    placement behavior).

Validation:
  basedpyright: 0 errors, 0 warnings, 0 notes.
  ruff check: clean.
  ruff format: clean.
  pytest: 923 passed, 1 skipped (was 921 + 2 new tests).

* Address Codex round-2 feedback on PR #20 bind-retry helper

Codex flagged two issues with the round-1 _bind_drafter_listener_with_retry:

P1 (utils_mlx.py:452): re-rolling the port broke the placement
contract. The drafter dials DrafterPlacement.drafter_socket_port,
which the master announced; switching the listener to a fresh
random port made the listener accept on a port the drafter never
tries, leaving the connection stalled until accept_drafter's 180s
timeout. The retry must keep the SAME port -- the realistic
EADDRINUSE case in cross-host deploys is TIME_WAIT residue from a
previous instance on the same port, which clears within ~100 ms.

P2 (utils_mlx.py:440): the retry caught every OSError, so non-
collision errors (EAFNOSUPPORT for an IPv6 wildcard on an IPv4-only
host, EACCES for a privileged port) got retried as if transient
and surfaced after the budget as a misleading "ephemeral port range
exhausted" error. Only EADDRINUSE is transient; everything else
must surface immediately so the operator sees the real root cause.

Changes:
  - utils_mlx.py: rename _bind_drafter_listener_with_retry to
    _bind_drafter_listener_same_port_retry; remove
    random_ephemeral_port re-roll; filter on errno.EADDRINUSE only;
    add 100ms sleep between retries for TIME_WAIT residue absorption.
    Final error message references the placement-announced port so
    the operator knows to re-place rather than guess at the bind
    address.
  - test_utils_mlx_bind_retry.py: 6 new unit tests covering
    same-port retry success, persistent EADDRINUSE budget exhaust,
    non-collision OSError immediate surface, and ordering when
    EADDRINUSE precedes a non-collision error.

Validation:
  basedpyright: 0 errors, 0 warnings, 0 notes.
  ruff check: clean. ruff format: clean.
  pytest: 929 passed, 1 skipped, 201 deselected, 0 failures (+6).

---------

Co-authored-by: jw-wcv <101585096+jw-wcv@users.noreply.github.com>